Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add migx ep fp8 int4 #78

Open
wants to merge 10 commits into
base: rocm6.3_internal_testing
Choose a base branch
from

Conversation

TedThemistokleous
Copy link

Description

Datatype support for int4 and fp8 (all formats)

Motivation and Context

Allows us to support operators of these data types to be handled by Onnxruntime MIGraphX EP

@TedThemistokleous TedThemistokleous added the enhancement New feature or request label Dec 4, 2024
@TedThemistokleous TedThemistokleous self-assigned this Dec 4, 2024
Map things to int8 right now as we don't explicitly set an int4 input type and pack/unpack int4 operands
Mirror the same calibration code we use for int8 and just change which quantize we call through the MIGraphx API
- Add additional flags for fp8 thats shared for int8

- Add lockout warning message when int8/fp8 used at the same time
Previous runs using session options failed as we were missing pulling in inputs from the python interface. This plus additional logging allowed me to track what options were invoked via env and what were added during the start of an inference session
need this so the user knows there's any of the environment variables running in the background to ensure proper consistently between runs.
@streamhsa streamhsa force-pushed the add_migx_ep_fp8_int4 branch from c19fa76 to d1a2609 Compare December 25, 2024 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant